This section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present invention that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
The problem of point and patch tracking is a widely studied and still open issue with implications in a broad area of computer vision and image processing. On one side and among others, applications such as object tracking, structure from motion, motion clustering and segmentation, and scene classification may benefit from a set of point trajectories by analyzing an associated feature space. In this case, usually a sparse or semi-sparse set of meaningful points needs to be tracked such as described by Sand and Teller in “Particle Video: Long-Range Motion Estimation Using Point Trajectories” (IJCV, vol. 80, no. 1, pp. 72-91, 2008). Indeed, those points that carry important information about the structure of the scene are more easily tracked. Recent approaches as those presented by Brox and Malik in “Object segmentation by long term analysis of point trajectories” (Proc. ECCV, 2010) or by Fradet, Robert, and Pérez in “Clustering point trajectories with various life-spans” (Proc. IEEE CVMP, 2011) are examples of the importance of long-term motion cues for spatio-temporal video segmentation.
On the other side, applications related to video processing such as augmented reality, texture insertion, scene interpolation, view synthesis, video inpainting and 2D-to-3D conversion eventually require determining a dense set of trajectories or point correspondences that permit to propagate large amounts of information (color, disparity, depth, position, etc.) across the sequence. Dense instantaneous motion information is well represented by optical flow fields and points can be simply propagated through time by accumulation of the motion vectors. That is why state-of-the-art methods as described by Brox and Malik in “Object segmentation by long term analysis of point trajectories” (Proc. ECCV, 2010) or by Sundaram, Brox and Keutzer in “Dense point trajectories by GPU-accelerated large displacement optical flow” (Proc. ECCV, 2010) have built on top of optical flow, methods for dense point tracking using such accumulation of motion vectors.
There are drawbacks to the methods for dense point tracking as mentioned above. In case of direct long-term estimation, the colour or the aspect of an object may change between 2 distant frames, thus leading to an imprecise motion field between the 2 frames. In the case of dense point tracking relying on accumulation, a drift in the displacement of the pixel may challenge the accuracy of the method.
The technical problem to solve is to provide an improved dense displacement map, also called motion field, between two frames of the video sequence.
The present invention provides such a solution.